Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction

The problems of object class segmentation [2], which assigns an object label such as road or building to every pixel in the image and dense stereo reconstruction, in which every pixel within an image is labelled with a disparity [1], are well suited for being solved jointly. Both approaches formulate the problem of providing a correct labelling of an image as one of Maximum a Posteriori (MAP) estimation over a Conditional Random Field (CRF). Both may use graph cut based move making algorithms to solve the labelling problem. The correct labelling of object class can inform depth labelling and stereo reconstruction can also improve object labelling. Similarly object class boundaries are more likely to occur at a sudden transition in depth and vice versa. Moreover, the height of a point above the ground plane is an extremely informative cue regarding its class label, and can be computed from the depth. For example road or sidewalk lie in the ground plane, and pixels taking labels pedestrian or car must lie above the ground plane, while pixels taking label sky must occur at an infinite depth. Our joint optimisation consists of two parts, object class segmentation and dense stereo reconstruction. We follow [2] in formulating the problem of object class segmentation as finding a minimal cost labelling of a CRF defined over a set of random variables X = {X1, . . . ,XN} each taking a state from the label space L = {l1, l2, . . . , lk}. Each label l j indicates a different object class such as car, road, building or sky. For the dense stereo reconstruction part of our joint formulation we use the energy formulation of [1], who formulated the problem as one of finding a minimal cost labelling of a CRF defined over a set of random variables Y = {Y1, . . . ,YN}, where each variable Yi takes a state from the label space D = {d1,d2, . . . ,dm} corresponding to a set of disparities. We formulate simultaneous object class segmentation and dense stereo reconstruction as an energy minimisation of a dense labelling z over the image. Each random variable Zi = [Xi,Yi] takes a label zi = [xi,yi], from the product space of object class and disparity labels L ×D that correspond to the variable Zi taking object label xi and disparity yi. In general the energy of the CRF for joint estimation can be written as: